fix(core): align assert-set scoring semantics - #1644

Summary

AgentV-authored type: assert-set YAML now matches Promptfoo's compatible scoring shape: omitted thresholds follow child pass/fail results, explicit thresholds gate on aggregate score, child weights affect aggregation, and parent metrics preserve grouped score names. Assert-set-level config is accepted in authored YAML and inherited by children with child config taking precedence, including for runtime script assertion context.

Semantic Decisions

• No-threshold assert sets pass only when all nonzero-weight children pass.
• Explicit threshold overrides child verdicts and gates on the weighted aggregate score.
• Child weight contributes to aggregate scoring; weight: 0 children are metric-only and do not fail an omitted-threshold parent.
• Parent metric names the assert-set aggregate score while child metrics remain visible as nested/named scores.
• Public YAML stays Promptfoo-shaped with nested assert; internal TypeScript keeps AssertSetGraderConfig.assertions because it is established AgentV API shape and clearer inside the grader boundary.

Promptfoo Evidence

Verified against local Promptfoo clone commit 6bfc5a0c7f16f9c4717ac731d276b578e63d0769.

• src/types/index.ts: AssertionSetSchema supports nested assert plus optional config, metric, weight, and threshold.
• src/assertions/assertionsResult.ts: no-threshold assertion sets follow child pass/fail results; thresholds override with aggregate score checks; child weights and named scores feed aggregation.
• src/assertions/index.ts: assert-set assertions are flattened through a parent assertion-set result, preserving parent metric/weight behavior.

Intentional Divergences

• AgentV treats explicit threshold: 0 as an authored threshold because AgentV threshold handling uses definedness, while the inspected Promptfoo code uses a truthiness check for threshold override.
• AgentV implements assert-set config inheritance for authored YAML and programmatic configs. The Promptfoo schema exposes set-level config, but the inspected flattening path did not visibly merge that config into children; AgentV's inheritance is the documented compatibility behavior for this surface.
• No public exported TypeScript names were churned solely for parity. The wire/authored format remains Promptfoo-compatible where intended.

Verification

• bun test packages/core/test/evaluation/loaders/grader-parser.test.ts packages/core/test/evaluation/graders/promptfoo-assertions.test.ts packages/core/test/evaluation/validation/eval-file-schema.test.ts
• bun --filter @agentv/core lint
• bun --filter @agentv/core typecheck
• bun --filter @agentv/core build
• bun --filter @agentv/sdk build
• bun --filter @agentv/web build
• bun run lint
• git diff --check

bun run validate:examples was not run because examples were not changed. The full root bun run test suite was not run; focused parser/schema/grader coverage was run for the touched behavior.

Dogfood And Evidence

Live dogfood used the local OpenAI-compatible proxy at http://127.0.0.1:10531/v1 with model gpt-5.4-mini, an OpenAI-compatible target, and a live llm-rubric child grader.

Command:

LOCAL_OPENAI_PROXY_BASE_URL=http://127.0.0.1:10531/v1 LOCAL_OPENAI_PROXY_API_KEY=local-proxy LOCAL_OPENAI_PROXY_MODEL=gpt-5.4-mini bun apps/cli/src/cli.ts eval run /tmp/agentv-av-mc8q-dogfood/assert-set-live.eval.yaml --targets /tmp/agentv-av-mc8q-dogfood/targets.yaml --target proxy-llm --workers 1 --threshold 0.5

Result: PASS (1/1 scored >= 50%, mean: 100%).

Run bundle: .agentv/results/2026-07-04T07-32-25-895Z.

Private evidence: EntityProcess/agentv-private branch evidence/av-mc8q-assert-set-semantics, commit 98bb85d.

Post-Deploy Monitoring And Validation

No additional operational monitoring is required for this local library/CLI scoring semantics change. CI should remain the merge gate; after merge, watch for core test/build/lint failures and user reports of assert-set score or verdict regressions.

Related

Related: av-mc8q